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ABSTRACT 

In  the  area  of  command  and  control  projects  where  a  surveillance  sensor  is  being  evaluated, 
project  sponsors  ask  scientists  to  measure  the  operational  utility  and  value  of  the  sensor  in  order 
to  assist  in  the  decision  making  process.  This  paper  puts  forward  a  formal  and  consistent 
methodology  of  relating  trial  results  to  operational  value-added  by  employing  the  Canadian 
Forces  Force  Planning  Scenarios.  The  approach  was  applied  within  an  Operational  Evaluation 
(OPVAL)  of  a  High  Frequency  Surface  Wave  Radar  (HFSWR)  system  that  is  used  for  over  the 
horizon  coastal  surveillance.  The  results  of  this  OPVAL  were  assessed  to  determine  HFSWR’ s 
contribution  to  surveillance-based  Force  Planning  Scenarios.  The  primary  contribution  of  this 
paper  is  to  demonstrate  the  feasibility  of  relating  OPVAL  results  directly  to  the  scenarios  in  a 
quantifiable  fashion.  The  main  conclusion  of  the  paper  is  that  the  approach  presents  the 
capabilities  of  the  sensor  system  to  the  decision  makers  in  a  logical  and  intuitive  way  and 
provides  a  context  to  the  results  of  the  analysis  that  is  easy  to  communicate  to  a  larger  audience. 
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INTRODUCTION 


The  Recognized  Maritime  Picture  (RMP)  is  defined  as  a  composite  picture  of  activity  of  a 
maritime  area  of  interest  for  a  given  time.  An  RMP  requires  timely  input  from  many  data 
sources  to  determine  the  location,  identity,  and  activity  of  targets  of  interest,  in  order  to  provide 
sufficient  information  to  decision  makers.  A  number  of  civilian,  military,  and  allied  sensor 
systems  contribute  to  the  development  of  an  RMP.  These  systems  include  automatic  self- 
reporting  positional  systems,  air  surveillance,  over-the-horizon  radar,  electronic  intelligence, 
space-based  radar,  and  high  frequency  direction  finding  sensors.  The  costs  of 
developing/acquiring  new  sensors  (or  even  including  sensor  reports  from  our  allies  or  other 
government  departments)  can  be  high.  Accordingly,  decision  makers  require  a  solid  basis  upon 
which  to  evaluate  potential  sensors  prior  to  making  sensor  procurement  or  development 
decisions. 

In  the  area  of  command  and  control  projects  where  a  surveillance  sensor  is  being  evaluated, 
project  sponsors  often  ask  scientists  to  measure  the  operationally  utility  of  the  sensor  in  order  to 
provide  justification  for  buying  one  sensor  suite  over  another.  Traditionally,  defence  scientists 
employ  Measures  of  Performance/Effectiveness  (MOP/MOEs)  to  determine  the  value-added 
provided  by  the  sensor  [Dickinson  et  al.,  1997]. 

The  authors  have  adopted  a  more  formal  and  consistent  methodology  of  evaluating  projects  to 
gain  acceptance  by  sponsors  and  stakeholders  and  thereby  improve  the  decision  making  process. 
This  approach  employs  the  Canadian  Forces  Force  Planning  Scenarios  (FPS),  which  are  based  on 
the  tasks  and  roles  identified  in  the  1994  Defence  White  Paper  [National  Defence,  1994].  These 
scenarios  are  representative  of  the  generic  types  of  missions  that  the  CF  has  been  called  upon  to 
perform.  Basing  evaluation  decisions  on  these  scenarios  authenticates  the  evaluation  process  and 
improves  the  odds  that  the  right  sensor  is  purchased  for  the  right  reasons. 

The  following  approach  was  taken  for  the  Operational  Evaluation  (OPVAF)  of  a  new  Canadian 
High  Frequency  Surface  Wave  Radar  (HFSWR)  system.  First,  the  scenarios  that  relate  to  the 
operational  role  of  HFSWR  were  identified  (e.g.  Search  and  Rescue,  Surveillance).  Second, 
representative  vignettes  were  developed  for  each  of  the  selected  scenarios  (e.g.  a  vessel  in 
distress,  contraband  carrying  vessel  entering  Canadian  waters).  Third,  MOP  were  devised  for 
each  vignette  (e.g.  positional  accuracy,  number  of  reports).  Fourth,  quantifiable  “value-added 
benefits”  were  developed  for  each  vignette  (e.g.  improved  positional  accuracy  and  accurate 
continuous  tracking  are  very  desirable  benefits  that  a  sensor  can  provide  to  SAR  and 
Surveillance  scenarios  respectively).  Fifth,  trial  data  from  the  HFSWR  OPVAF  was  used  to 
determine  whether  HFSWR  could  provide  these  value-added  benefits,  and  to  what  extent. 


The  authors  demonstrate  both  the  utility  and  drawbacks  of  this  approach  in  assessing 
surveillance  sensors.  This  is  done  using  OPVAL  trial  results  from  the  HFSWR  system  located 
at  Cape  Race,  Newfoundland.  Independent  “ground-truth”  was  obtained  on  a  cooperative 
controlled  ship.  The  HFSWR  data  was  assessed  for  its  contribution  to  the  RMP,  both 
individually  and  collectively  (as  fused  data),  given  the  requirements  of  the  FPS  vignette. 

2.  HF  SURFACE  WAVE  RADAR  SYSTEM  (HFSWR) 

The  following  section  describes  the  HFSWR  system  and  discusses  an  OPVAL  trial  that  was 
conducted  in  Jan  2002.  The  results  of  this  OPVAL  are  used  as  input  into  this  Scenario-Based 
Assessment  and  will  be  presented  in  Section  3. 

2.1  HFSWR  TRACKS 

HFSWR  is  a  potential  data  source  that  is  being  seriously  considered  for  acquisition  by  the 
Department  of  National  Defence  (DND).  HF  surface  waves  propagate  along  the  ocean  surface 
and,  because  of  diffraction,  follow  the  curvature  of  the  earth.  HFSWR  can  achieve  long  ranges 
due  to  low  attenuation  rates  of  vertically  polarized  HF  electro-magnetic  signals  when  propagated 
over  the  conductive  ocean  surface.  There  are  presently  two  HFSWR  sites  located  in 
Newfoundland,  one  at  Cape  Race  and  the  other  at  Cape  Bonavista,  that  are  being  used  to 
demonstrate  the  capability  of  the  HFSWR  technology  [Moutray  and  Ponsford,  1997].  The 
system  employs  an  array  of  antennas  with  coherent  receivers  at  the  output  of  the  antenna 
elements.  The  signal  is  sampled  and  the  required  filtering,  beam  forming,  Doppler  analysis, 
tracking,  noise  suppression  and  other  signal  processing  operations  are  performed.  In  order  not  to 
overload  the  command  and  control  system  with  false  detections,  only  the  output  of  the  tracker  is 
sent  for  inclusion  in  the  RMP. 

The  HFSWR  system  is  being  considered  for  surveillance  out  to  the  200  NM  Exclusive  Economic 
Zone  (EEZ).  The  system  provides  a  number  of  advantages  relative  to  other  wide  area  maritime 
surveillance  systems:  it  is  relatively  inexpensive  to  install  and  operate,  it  operates  virtually 
continuously,  and  the  tracker  output  provides  near-real  time  vessel  position,  course,  and  speed. 
On  the  other  hand,  HFSWR  must  operate  in  a  congested  signal  spectrum  with  limited  signal 
bandwidth,  in  the  presence  of  sea  clutter,  ionospheric  interference  and  both  external  and  man¬ 
made  noise. 


The  HFSWR  tracker  is  currently  optimized  for  non-manoeuvring  targets  since  transiting  vessels 
will  maintain  a  straight  course,  or  leg,  for  long  periods  of  time.  The  majority  of  the  vessels  in  the 
area  of  interest  will  maintain  a  time-on- leg  (TOL)  of  greater  than  4  hours.  However,  the  tracker 
tends  to  produce  broken  tracks  with  unique  identification  numbers  when  the  vessel  is 
manoeuvring.  An  example  of  a  Canadian  Patrol  Frigate  being  tracked  during  a  trial  at  relatively 
close  range  is  shown  in  Figure  1.  Each  time  the  vessel  turns  the  tracker  initiates  a  new  track  and 
assigns  a  new  track  identification  number. 


Range  rings 

. Vessel  GPS  position 

X  -  HFSWR  track 


Figure  1 :  HFSWR  tracking  of  vessel  from  Cape  Race  site  with  ground  truth 


Existing  data  fusion  software  (e.g.  [Lefebvre  et  al,  2001]  and  [Simard  el  al,  2000])  provides  a 
capability  that  will  associate  track  segments  if  the  manoeuvre  is  not  very  abrupt  and  when  other 
data  sources  provide  track-to-track  fusion  potential.  This  type  of  software  works  well  for 
transiting  vessels  making  corrections  to  their  course  but  is  ineffective  when  the  vessel  is 
manoeuvring  abruptly  and  frequently.  Fishing  vessels,  which  may  have  small  radar  cross- 
sectional  area,  are  constantly  manoeuvring  while  fishing,  thereby  resulting  in  low  average  time- 
on-leg  values.  As  a  result  they  often  produce  broken  tracks. 


The  decision  to  acquire  the  system  is  based  on  the  results  of  an  OPVAL  that  determines  the 
value-added  utility  of  HFSWR  to  the  RMP.  The  evaluation  process  is  further  complicated  by  the 
fact  that  any  data  source  must  be  interoperable  with  other  surveillance  systems  to  generate  an 
RMP. 

2.2  HFSWR  OPVAL  TRIAL 

Controlled  Ship  Trial  Jan  4-7,  2002 

A  trial  involving  a  controlled  ship  was  conducted  within  the  nominal  coverage  region  of  the 
HFSWR  system  situated  at  Cape  Race,  Newfoundland.  Data  was  collected  from  HFSWR  and 
compared  against  the  position  of  the  ship  as  it  followed  its  pre-determined  route.  Members  of 
the  crew  manually  recorded  ship  positional  data  at  a  minimum  of  every  15  minutes  throughout 
the  60-hour  trial. 

Seven  Measures  of  Performance  (MOP)  concepts  were  used  to  assess  the  HFSWR  performance. 
A  brief  description  of  these  is  provided  in  Table  I. 

Table  I:  Relevant  MOP  Concepts  for  an  ISR  System  Evaluation 

Description 

The  total  number  of  individual  detections  from  a  surveillance  source,  which  contribute 
to  building  up  the  RMP. 

The  distance  between  the  actual  position  and  that  reported  by  the  sensor  at  a  given  time. 

The  ability  of  a  source  to  provide  information  that  rules  out  and/or  helps  to  identify  a 
vessel  of  interest.  The  longer  a  system  is  able  to  maintain  the  same  track  ID  number,  the 
lower  the  track  ambiguity.  Conversely,  the  more  different  track  ID  numbers  associated 
with  one  vessel,  the  higher  the  track  ambiguity. 

This  considers  of  two  main  variables:  first,  the  period  of  time  between  successive 
contact  reports  and  second,  the  time  between  the  initiation  of  a  contact  report  and  its 
arrival  into  the  RMP. 

The  area  within  which  the  system  can  make  reports  on  vessels. 

The  ability  of  the  surveillance  source  to  pick  up  changes  in  direction  and  speed  of 
vessels  tracked  and  its  ability  to  detect  and  track  small  vessels. 

This  assesses  the  ability  of  the  system  to  provide  value  when  working  together  with 
other  sensors  (e.g.  the  existing  RMP  sensors).  Uniqueness  describes  the  condition  where 
a  target  vessel  is  detected  by  only  one  sensor,  whereas  Complementarity  describes  the 
condition  where  more  than  one  sensor  is  in  contact  with  the  vessel  (the  mathematical 
complement  of  the  number  of  unique  contacts). 


MOP 

Number  of  Reports: 

Positional  Accuracy: 
Track  Ambiguity: 

Timeliness: 

Coverage: 

Sensitivity: 


Uniqueness  and 
Complementarity: 


Assessment  ofHFSWR  Performance 


Extensive  analysis  was  performed  on  the  data  that  was  collected  from  this  60  hour  OPVAL  trial 
though  only  the  salient  results  are  presented  here.  Note  in  Figure  2,  that  HFSWR  tracking  is 
shown  in  shades  of  blue  while  the  ground  truthed  track  of  the  ship  for  all  ten  legs  is  marked  with 
grey  circles.  For  the  first  six  legs  of  the  route,  positional  updates  of  the  ship  were  provided  every 
15  minutes.  For  legs  7  through  10  inclusive,  these  updates  were  performed  every  five  minutes. 


Figure  2.  HFSWR  OPVAL  Trial.  Controlled  ship  track  (grey  circles)  and  HFSWR  tracks  (blue). 

HFSWR  tracked  the  ship  around  the  route  using  six  different  track  identification  numbers. 
HFSWR’ s  maximum  detection  range  suffered  because  of  the  environmental  conditions  during 
the  trial  (i.e.  high  levels  of  ionospheric  and  surface  clutter),  but  when  it  was  able  to  track  the 
vessel,  the  system  gave  timely  and  accurate  tracking  information.  On  its  own,  HFSWR  was  able 
to  detect  and  track  the  vessel  during  53%  of  the  controlled  ship  route  (when  operating  inside  the 
HFSWR  coverage  area)  and  when  working  with  other  RMP  sensors,  the  composite  track  was 
increased  to  78%. 

\*  MERGEFORMAT  During  this  OVPAF  trial,  HFSWR  excelled  in  three  of  the  seven  MOPs, 
performing  significantly  better  than  the  other  data  sources  currently  contributing  to  the  RMP. 


a.  Number  of  Reports.  HFSWR  provided  373  reports  on  the  controlled  ship  over  a  60- 
hour  period,  which  was  significantly  higher  than  the  other  existing  sensors. 

b.  Timeliness.  HFSWR  reported  a  new  position  every  4.3  minutes;  and  the  data  was 
received  in  the  RMP  within  5  minutes  of  detection.  This  was  much  better  than  other 
sources. 

c.  Uniqueness  and  Complementarity.  The  benefit  added  to  the  RMP  by  both  the 
uniqueness  and  the  complementarity  of  the  HFSWR  tracks  was  significant,  improving 
the  RMP  by  more  that  100%  when  HFWR  was  added  to  the  main  RMP  data  source. 

With  respect  to  the  remaining  MOP,  HFSWR  provided  value-added  benefits  that  ranged  from 
minimal  (e.g.  ‘Sensitivity’)  to  moderate  (e.g.  ‘Positional  Accuracy’)  improvement  relative  to 
existing  RMP  sensors. 


3.  FORCE  PLANNING  SCENARIOS  BASED  DECISION  MAKING 

The  evaluation  above  provides  a  valuable  assessment  of  the  overall  utility  of  the  HFSWR  system 
to  the  generation  of  the  RMP,  but  it  does  not  provide  the  decision  maker  with  a  context  in  which 
to  make  a  decision  whether  to  utilize  the  capability  or  to  further  develop  it.  The  authors  are 
proposing  that  evaluating  the  benefit  provided  by  the  system  in  recognized  scenarios  can  provide 
this  “real  world”  context. 

The  Vice  Chief  of  the  Defence  Staff  (VCDS)  has  published  a  comprehensive  set  of  eleven  Force 
Planning  Scenarios  (FPS)  for  strategic  planning.  This  section  of  the  report  introduces  the 
experimental  technique  devised  by  the  authors  that  compares  the  utility  of  HFSWR  against  the 
most  relevant  FPS  relating  to  Intelligence  Surveillance  and  Reconnaissance  (ISR).  This 
approach  was  developed  to  address  the  lack  of  a  fully  developed  Operational  Statement  of 
Requirement  for  ISR,  which  is  one  of  the  major  challenges  of  this  OPVAL  process. 


3.1  METHODOLOGY 

The  objective  of  the  proposed  methodology  is  to  provide  a  link  between  the  relevant  aspects  of 
the  scenarios  and  the  performance  measures.  The  authors  believe  that  this  methodology  can  be 
used  for  any  ISR  system  being  evaluated  for  inclusion  into  the  RMP. 

The  process  is  outlined  in  the  Steps  described  below.  The  results  are  summarized  for  the 
HFSWR  system  evaluation  in  Table  II,  III,  and  IV. 


Stepl: 

Step2: 

Step  3: 
Step  4: 

Step  5: 

Step  6: 


Select  the  scenarios  for  which  the  sensor  system  is  potentially  relevant  (i.e.  scenarios 
that  have  a  Canadian  ISR  flavour).  (Column  1  of  Table  II) 

Produce  maritime  vignettes  (scripted  events)  that  best  represent  each  ISR  scenario 
that  would  involve  the  sensor  under  consideration.  (Column  2  of  Table  II) 

Determine  the  objectives  of  ISR  for  each  scenario/vignette.  (Column  3  of  Table  II) 

Identify  five  value-added  benefits  for  each  scenario  that  best  match  the  objectives  of 
Table  II  (Column  2  of  Table  III). 

Score  the  benefits  using  the  method  described  in  Section  3.2  while  considering  the 
OPVAL  trial  results.  (Column  3  of  Table  IV). 

Provide  a  concise  justification  of  the  scores  provided  (Column  4  of  Table  IV). 


The  first  three  steps  require  interaction  and  collaboration  with  the  stakeholders  to  ensure  the 
accuracy  of  any  underlying  assumptions  and  to  gain  their  acceptance  of  this  methodology.  For 
this  OPVAL,  four  VCDS  scenarios  were  selected  that  require  significant  ISR  capability:  (Search 
and  Rescue  (SAR),  Surveillance,  National  Sovereignty,  and  Defence  of  North  America).  As 
these  scenarios  are  too  general  in  nature  to  provide  a  straightforward  evaluation  of  the  sensor 
system;  maritime  vignettes  were  produced  for  each  of  the  four  ISR  scenarios  and  a  set  of  ISR 
objectives  was  established  for  each  vignette.  The  objectives  provide  a  means  to  evaluate  the 
sensor  system  in  the  context  of  the  scenario/vignette.  Table  II  illustrates  the  results  of  Steps  1,  2 
and  3  for  the  HFSWR  OPVAL. 


Table  II.  Maritime  ISR  Scenarios  &  Objectives 


ISR  FPS 

Maritime  Vignettes 

ISR  Objectives 

Search  and 
Rescue  (SAR) 

Initiated  by  distress  signal  - 
general  location  of  search  area 
provided 

•  Identify  SAR  vessel 

•  Establish  accurate  datum  position  and  time 

•  Track  SAR  vessel  and  direct  rescuing  unit 

SAR  vessel  has  been  reported 
missing  with  a  route  but  no 
specific  location  provided 

•  Define  problem  and  reduce  search  area 

•  Develop  the  profile  of  vessel  (capabilities  and 
intentions) 

•  Establish  an  Area  of  Probability  (AOP) 

•  Establish  track  history  to  aid  in  search 

Surveillance 

Contraband 

(All  smuggling-  drugs, 
people  etc) 

Pollution 

Identity  of  smuggling  vessel,  the 
departure  port,  and  time  of 
departure  is  known  as  well  as  the 
general  destination  of 
contraband  carrying  ship. 

•  Establish  AOP 

•  Establish  contact  box  (high  confidence  that  Vessel 
of  Interest  (VOI)  is  inside) 

•  Reduce  ambiguity  by  disregarding  vessels  on  the 
basis  of  Course,  speed,  and  origin 

•  Maintain  a  track  on  VOI 

Intelligence  provides  description 
of  contraband  activity  but  no 
identification  or  route  plan 

•  Intel  focus  -  devise  surveillance  plan  &  AOP 

•  Identify  suspicious  activity 

A  pollution  slick  has  been 
sighted  but  no  ID  of  polluter 

•  Establish  track  history  to  pollution  site 

•  Collect  evidence,  report  to  authorities 

National 

Sovereignty 

Monitoring  illegal  fishing 
activities  inside  EEZ  in  vicinity 
of  the  Nose  and  Tail  of  the  Bank 

•  Detect  presence  and  level  of  activity  inside  EEZ 

•  Develop  profile  of  VOI  (capabilities  and  intentions) 

Defence  of 

North  America 

Hostile  act  has  been  committed; 
prosecution  of  attacker  is 
conducted  after  the  fact 

•  Obtain/maintain  a  track  on  VOI 

•  Collect  evidence  and  report  to  responding  unit 

On-going  incident  (hostage  etc) 

•  Establish  an  AOI  and  report  to  authorities 

•  Obtain  and  maintain  situational  awareness 

•  Identify  combatants/non-combatants 

Attempt  to  prevent  extremist 
activity  with  Intel,  no  ID  of 
hostile  forces 

•  Define  the  problem  (W5  +  How) 

•  Establish  profile  of  vessel  (capabilities  and 
intentions,  etc.)  &  AOP 

•  Identify  combattants  /  non-combattants 

Operational  support  to  a 
Naval/Joint  operation 

•  Provide  situation  awareness  outside  of  ship/organic 
air  assets  sensor  range 

3.2  SCORING  TECHNIQUE  1 

This  scoring  technique  was  developed  to  quantify  the  potential  value-added  benefits  that 
HFSWR  could  bring  to  the  RMP.  From  the  assessment  made  on  the  HFSWR  system,  it  is 
possible  to  identify  the  system’s  key  strengths.  To  rate  these  and  analyze  them  against  the  ISR 
scenarios  requires  choosing  a  suitable  scoring  mechanism. 

Assigning  an  appropriate  score  to  each  attribute  of  a  system  is  not  an  exact  science  and  it  is  a 
challenging  task.  Many  researchers  have  written  about  the  inherent  difficulty  in  capturing  the 
decision  making  process  by  weighting  attributes  and  creating  scoring  systems.  Most  admit  that 
there  is  a  level  of  subjectivity  that  must  be  injected  into  the  process  ([Keeney  and  Raiffa,  1976] 
[Zeleny,  1982]).  Nevertheless,  the  essence  of  a  scoring  technique  is  to  capture  the 
importance/value  of  each  attribute  of  a  system  and  generate  a  score,  index  or  rating  scale  that 
will  assist  the  decision  maker  in  understanding  the  value  or  benefit  of  any  grouping  of  attributes. 

For  the  purpose  of  this  scenario-based  technique,  it  was  important  to  choose  a  scoring  system 
that  would  capture  what  was  observed  during  the  OPVAL  trial  and  condense  this  information 
down  to  one  score  that  could  be  applied  to  each  ISR  scenario. 

Referring  to  the  MOP  analysis  in  Section  2.2,  HFSWR  exhibited  a  significant  improvement  over 
the  existing  systems  in  the  RMP  for  three  measures  and  moderate  improvement  or  less  for  the 
remaining  four  measures.  In  order  to  capture  these  observations,  the  authors  chose  to  adopt  a 
scoring  system  for  each  objective  in  the  selected  ISR  scenarios.  For  each  objective,  HFSWR 
was  assessed,  based  on  the  analysis  of  its  performance  in  the  trials,  according  to  Table  III. 


1  The  authors  would  like  to  thank  Mr.  Ed  Emond  (DND-CORA/CORT)  for  his  invaluable  contribution  to  the 
development  of  this  scoring  technique. 


Table  III.  Scoring  Method 


Score 

Substantiation 

0.0 

HFSWR  demonstrated  no  improvement  to  the  RMP 

0.1 

HFSWR  demonstrated  a  slight  to  moderate  improvement 

1.0 

HFSWR  demonstrated  a  significant  improvement  to  the  RMP  (ie.  ‘a  quantum  leap'  above  all  other 
existing  systems) 

This  scoring  method  had  several  favorable  qualities: 


a.  it  summarizes  the  value  of  significant  contributions  by  HFSWR  and  contrasts  them 
with  respect  to  areas  of  slight  to  moderate  improvement  and  no  improvement, 

b.  it  presents  an  additive  scoring  system,  where  every  scenario-based  objectives  are 
considered, 

c.  it  provides  a  single  score  for  HFSWR  for  each  scenario,  presenting  to  the  decision 
makers  a  single  number  that  is  clear  and  easy  to  interpret,  and 

d.  it  provides  a  mechanism  that  can  be  used  to  compare  other  ISR  systems  against  each 
other. 

Note  in  Table  IV,  that  five  value-added  benefits  were  determined  for  each  scenario.  This  was 
purposely  done  so  that  the  scores  would  add  up  to  a  maximum  score  of  5.  This  is  not  an 
essential  step  and  it  is  possible  to  have  any  number  of  benefits  per  scenario.  However,  having  a 
common  scoring  base  simplifies  the  comparison  and  makes  the  results  easier  to  interpret  by  the 
decision  maker. 

4.  ISR  FORCE  PLANNING  SCENARIO  ANALYSIS 

4.1  HFSWR  FPS  CONTRIBUTION 

Applying  our  scoring  system  to  the  value-added  benefits  defined  in  Table  III  produced  the  results 
shown  in  Table  IV.  A  score  was  determined  for  each  value-added  benefit  based  on  the  results 
from  the  trial.  For  example,  consider  the  SAR  scenario  value-added  benefit  ‘Reduce  time  need 
to  assist’  and  its  score  of  ‘1’.  As  there  are  no  other  near  real-time  sensors  available  for  the 
generation  of  the  RMP,  the  ability  of  the  HFSWR  system  to  provide  near  real-time  updates 
represents  a  significant  improvement  over  the  RMP  currently  produced  by  other  data  sources. 

During  the  trial  the  controlled  vessel  was  detected  out  to  a  maximum  range  of  140  nautical  mile 
(NM),  short  of  the  nominal  220  NM  range  limit,  due  to  extremely  high  levels  of  ionospheric 
interference.  Accordingly,  on  the  basis  of  this  single  trial  HFWSR  system,  it  was  assessed  that 
HFSWR  provided,  at  best,  moderate  improvement  for  the  value-added  benefit  ‘Detection  of  all 
vessels  in  an  area’  depending  on  the  scenarios.  For  SAR  and  Defence  of  North  America 
scenarios,  the  coverage  area  of  interest  is  anywhere  within  in  the  HFSWR  coverage  area; 
therefore,  the  partial  coverage  provided  by  HFSWR  was  good  enough  to  earn  a  0.1  score. 


However,  for  the  Surveillance  scenario  that  focuses  on  the  monitoring  of  illegal  foreign  fishing 
at  the  Nose  and  Tail  of  the  Grand  Banks  at  the  200NM  EEZ,  the  score  received  was  ‘0’  since 


HFSWR,  on  its  own,  provided  no  value  beyond  the  140  mile  range. 

Table  IV  HFSWR  Evaluation 


ISR  FPS 

Value  Added  Benefits 

HFSWR 

Score 

Remarks  &  Assessment  of  Improvement  due  to 
HFSWR 

SAR 

Increase  positional  accuracy 

0.1 

Moderately  better  positional  accuracy  ( 1.5NM) 

Detect  all  vessels  in  an  area 

0.1 

Detected  controlled  ship  53%  of  the  time  due  to 
range  shortfall  (140NM  vice  220NM) 

Reduce  time  need  to  assist 

1 

Only  near-real  time  sensor,  4.3  min  updates 

Identify  vessel 

0 

Doesn’t  provide  vessel  ID 

Accessible  data  to  third  party 

1 

Unclassified,  direct  feed 

SAR  Capability  improvement 

2.2 

Surveillance 

Detect  all  vessels  in  the  AOP 

0.1 

Detected  controlled  ship  53%  of  the  time  due  to 
range  shortfall  ( 140NMvice  220NM) 

Accurate  continuous  tracking 

1 

The  best  tracking  sensor  in  the  RMP 

Minimal  use  of  tasked  resources 

1 

Much  improved  RMP  due  to  uniqueness  and 
Complementarity  with  other  sources 

Identify  vessel 

0 

Doesn’t  provide  vessel  ID 

Correlate  contacts  in  the  area 

.1 

Good  tracking  helped  reduce  RMP  ambiguity 

Surveillance  Improvement 

3.2 

National 

Sovereignty 

Illegal 

Foreign 

Fishing 

(Nose  /  Tail 
of  the  Bank) 

Detect  violators 

0 

Could  not  detect  vessels  past  140NM 

Collect  irrefutable  evidence 

0 

Could  not  detect  vessels  past  140NM 

Minimal  use  of  tasked  resources 

0 

Could  not  detect  vessels  past  140NM 

Accessible  data  to  third  party 

0 

Unclass,  direct  feed  but  outside  of  range 

Direct  authorities  to  location 

0 

Could  not  detect  vessels  past  140NM 

Sovereignty  Capability  improvement 

0.0 

Defence  of 

North 

America 

Covert  tracking 

0.1 

HFSWR  operating  24/7 

Tracking  continuously 

1 

The  best  tracking  sensor  in  the  RMP 

Detect  all  vessels  in  an  area 

0.1 

Detected  controlled  ship  53%  of  the  time  due  to 
range  shortfall  (140NM  vice  220NM) 

Accessible  data  to  third  party 

1 

Unclassified,  direct  feed 

Identify  vessel 

0 

Doesn’t  provide  vessel  ID 

Defence  of  North  America  improvement 

2.2 

4.3  SUMMARY  REMARKS 


Comparing  HFSWR  within  the  ISR  FPS  context  injects  a  real  world  component  into  the  process. 
Based  on  the  observations/results  from  Table  IV,  HFSWR  provided  improvement  in  three  of  the 
four  ISR  scenarios.  The  FPS  showing  the  greatest  improvement  was  Surveillance;  achieving 
three  major  benefits  over  the  status  quo  and  two  minor  ones.  The  next  most  noticeable 
improvement  was  for  FPS  Defence  of  North  America  and  SAR;  where  two  major  benefits  and 
two  minor  benefits  were  noted.  The  FPS  for  National  Sovereignty  (illegal  foreign  fishing), 
however,  did  not  receive  additional  benefit  from  using  HFSWR  due  to  its  inability  to  track 
vessels  at  the  200NM  EEZ. 

These  scorings  are  based  on  a  single  three-day  trial.  Extremely  high  levels  of  ionospheric 
interference  characterized  the  trial  and  the  high  sea  states  produced  high  levels  of  surface  clutter. 
Results  from  more  extensive  trials  at  different  times  of  the  year,  different  weather  conditions  and 
vessels  with  significantly  different  radar  cross  section  areas  will  undoubtedly  produce  different 
results  that  could  alter  the  scoring.  Notwithstanding,  it  would  be  an  easy  and  straightforward 
process  to  re-score  HFSWR. 

The  strength  of  this  scenario  based  assessment  of  sensors  is  that  it  compares  performance  against 
actual  tasks  and  roles  outlined  in  the  Defence  White  Paper  and  provides  a  quantifiable 
performance  score  that  can  be  easily  communicated  to  decision  makers.  The  weaknesses  of  this 
approach  lie  in  its  subjectivity.  To  mitigate  these  weaknesses,  it  is  strongly  recommended  that 
stakeholders  are  involved  early  to  determine  and  prioritize  the  value-added  benefits  and  provide 
guidance  in  devising  a  scoring  system.  Once  this  is  done,  the  authors  believe  that  this  approach 
could  be  useful  and  easily  applied  when  comparing  a  broader  range  of  surveillance  sensors. 

5.  CONCLUSIONS 

In  this  paper  we  have  proposed  a  methodology  for  evaluating  the  value-added  benefit  of  a 
surveillance  system  to  the  generation  of  the  recognized  maritime  picture.  The  approach  provides 
a  direct  link  between  the  more  technical  aspects  of  an  OPVAL  and  the  decision  maker’s  need  for 
a  straightforward  measure  of  utility  through  the  use  of  the  Canadian  Forces  FPS.  These 
scenarios  are  representative  of  the  generic  types  of  missions  that  the  CF  may  be  called  upon  to 
perform.  Basing  evaluation  decisions  on  these  scenarios  authenticates  the  evaluation  process  and 
improves  the  odds  that  the  right  sensor  is  acquired  for  the  right  reasons.  The  approach  has  been 
applied  to  data  obtained  during  trials  of  the  HFSWR  system  located  off  the  east  coast 
Newfoundland,  Canada. 


The  approach  consists  of  determining  FPS  that  relate  to  the  operational  role  of  the  sensor  being 
evaluated;  developing  representative  vignettes  for  each  of  the  FPS;  tailoring  MOP  for  each 
vignette;  matching  trial  data  to  the  vignettes;  and  analyzing,  the  results  were  against  quantifiable 
measures  of  value.  The  benefits  and  drawbacks  of  this  approach  in  assessing  sensors  in  the 
context  of  ISR  have  been  demonstrated.  This  is  done  using  the  results  of  trial  data  for  HFSWR 
and  other  surveillance  sensors  that  were  detecting  targets  within  the  HFSWR  coverage  area.  The 
observed  tracks  were  matched  to  a  specific  vignette  profile  and  all  the  sensor  data  was  then 
assessed  for  their  contribution  to  the  RMP,  both  individually  and  collectively  as  fused  data,  given 
the  requirements  for  the  FPS  vignette. 

The  primary  contribution  of  this  paper  is  to  demonstrate  the  feasibility  of  relating  the  results 
from  an  operational  evaluation  to  the  Force  Planning  Scenarios  in  a  quantifiable  fashion.  The 
main  conclusion  of  the  paper  is  that  the  vignettes  present  the  capabilities  of  the  sensors  to  the 
decision  makers  in  a  logical  and  intuitive  way  and  they  give  a  context  to  the  results  of  the 
analysis  that  becomes  easy  to  communicate  to  the  larger  audience. 
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